arXiv:hep-ph/9808240v2 6 May 1999 


DFTT 46/98 
^ep-ph/98U824q 


A new ordering principle for the classical statistical analysis of 
Poisson processes with background 


C. Giunti 

INFN, Sezione di Torino, and Dipartimento di Fisica Teorica, Universitd di Torino, 

Via P. Giuria 1, 1-10125 Torino, Italy 


Abstract 


Inspired by the recent proposal by Feldman and Cousins Q of a “unified ap¬ 
proach to the classical statistical analysis of small signals” based on a choice 
of ordering in Neyman’s construction of classical confidence intervals, I pro¬ 
pose a new ordering principle for the classical statistical analysis of Poisson 
processes with background which minimizes the effect on the resulting con¬ 
fidence intervals of the observation of less background events than expected. 
The new ordering principle is applied to the calculation of the confidence re¬ 
gion implied by the recent null result of the KARMEN neutrino oscillation 
experiment |^. 
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I. INTRODUCTION 


In a very interesting paper []^ Feldman and Cousins have recently proposed a “unified 
approach to the classical statistical analysis of small signals” that allows to construct clas¬ 
sical confidence belts which “unify the treatment of upper confidence limits for null results 
and two-sided confidence intervals for non-null results”. This unified approach is very at¬ 
tractive because the transition from two-sided intervals to upper limits is automatic and 
the undercoverage introduced by a choice based on the data (“flip-flopping”) is avoided by 
construction. Indeed, the unified approach has promptly been adopted by the Particle Data 
Group as the new standard method for statistical inference |0]. 

The probability to observe a number n of events in a Poisson process consisting in signal 
events with mean p and background events with known calculated mean b is 

F(n|p,6) = ^ + (1.1) 

nl 

The classical method for obtaining the confidence interval for the unknown parameter /i is 
based on Neyman’s method to construct a confidence belt. This confidence belt is the region 
in the /i-n plane lying between the two curves a) and n 2 (p, a) such that for each value 
of fi 


P{n e [ni(p, a),n 2 (/i, a)]|/i, 6) = a, (1.2) 

where 

n2 

P{n e [ni,n 2 ]|/i, 6) = (1-3) 

n=n\ 

and a is the desired confidence level. The two curves ni(/i, a) and n 2 (/i, a) are required to 
be monotonic functions of /i and can be inverted to yield the corresponding curves /ii(n, a) 
and ^ 2 {n,a). Then, if a number nobs of events is measured, the confidence interval for fi is 
[/i 2 (?^obs,«),/^i(’^obs 5 «)]• This method guarantees by construction the correct coverage, i.e. 
the fact that the resulting confidence interval [/i 2 (?^obs,«),/Wi(nobs, a)] is a member of a set 
of confidence intervals obtained with an ensemble of identical experiments that contain the 
true value of fi with a probability a. Actually, in the case of a Poisson process, since n is 
an integer, the relation ( |1.2|) can only be approximately satisfied and in practice the chosen 
acceptance intervals [ni{fi,a),n 2 {fi,(y)] are the smallest intervals such that 

P{n e [ni{fi,a),n 2 {fi,a)]\fi,b) > a. (1.4) 

This choice introduces an overcoverage for some values of fi and the resulting confidence 
intervals are conservative. As emphasized in Ref. JI|, conservativeness is an undesirable 
but unavoidable property of the confidence intervals in the case of a Poisson process (it 
is undesirable because it implies a loss of power in restricting the allowed range for the 
parameter fi). 

The construction of Neyman’s confidence belt is not unique, because in general there 
are many different couples of curves ni(/i, a) and n 2 {fi,a) that satisfy the relation (p..2D. 
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Hence, an additional criterion is needed in order to define uniquely the acceptance intervals 
[ni(/i, a), n 2 (/i, a)]. The two common choices are 

1 — 01 

P{n < ni(/i, a)|/i, h) = P{n > n 2 (/x, a)|p, h) = —^— , (1.5) 

which leads to central confidence intervals and 

P{n < ni{fi,a)\fi,b) = 1 — a , (1.6) 


which leads to upper confidence limits. 

Feldman and Cousins proposed an ordering principle based on likelihood ratios that 
produces an automatic transition from a central confidence interval to an upper limit when 
the number of observed events in a Poisson process with background is of the same order or 
less than the expected background. The acceptance interval for each value of /i is calculated 
assigning at each value of n a rank obtained from the relative size of the ratio 


R{n) 




(1.7) 


where //best = hbest(^) (for a fixed b) is the non-negative value of pi that maximizes the 
probability P(n|/r, 6): 


Aibest(n) = max[0, n - b]. 


( 1 . 8 ) 


As emphasized in Ref. [Q , “i? is a ratio of two likelihoods: the likelihood of obtaining n given 
the actual mean pi, and the likelihood of obtaining n given the best-fit physically allowed 
mean”. The rank of each value of n is assigned in order of decreasing value of the ratio R{n): 
the value of n which has bigger R{n) has rank one, the value of n among the remaining ones 
which has bigger R{n) has rank two and so on. The acceptance interval for each value of /i 
is calculated by adding the values of n in increasing order of rank until the condition (p..4|) 
is satisfied. 

It is clear that the Feldman and Cousins ordering principle guarantees an automatic 
transition from two-sided confidence intervals to upper confidence limits for n < 6. Indeed, 


since pihestin < &) = 0, the rank of n < 6 for /i = 0 is one, implying that the interval 
0 < n < 6 for /i = 0 is guaranteed to lie in the confidence belt. 

Although the unified approach solves brilliantly the problem of obtaining a transition 
with correct coverage from two-sided confidence intervals to upper confidence limits for 
n < 6, it has the undesirable feature that even when n < b the upper bound p.\{n,a) 
decreases when b increases. From a physical point of view this is rather disturbing, because 
a stringent upper bound for p, obtained by an experiment which has observed a number of 
events significantly smaller than the expected background is not due to the fact that the 
experiment is very sensitive to small values of p, but to the fact that less background events 
than expected have been observed. Hence, two different experiments observing the same 
Poisson process with mean p and measuring the same number of events riobs but expecting 
a different value for the background b lead to two different upper limits for p and the upper 
limit established by the experiment that expect a larger background can be significantly 
smaller than the other one. This is illustrated in Figs.|^ and ^ 
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In Fig. [| I have plotted the 90% CL conhdence belts obtained with the unihed approach 
for b = 2,3,4, 5. One can see that four experiments expecting, respectively, 2, 3, 4 and 5 
background events and observing, for example, riobs = 1 lead to the 90% CL upper conhdence 
limit Pi = 2.53,1.88,1.34,1.20, respectively. 

In Fig. ^I have plotted the upper end pi of the 90% CL conhdence intervals obtained with 
the unihed approach for n = 0, 1,2, 3,4, 5 as a function of the mean expected background 
b. One can see that for each hxed value of n the upper end of the conhdence intervals 
tends to decrease rapidly as the mean expected background b increases for small values of 
b and stabilizes around a value close to 0.8 for large values of b. The decrease of pi is not 
monotonic, because of the discreteness of the observable n. Figure ^ dihers from Fig.8 of 
Ref. |jl[| for the fact that I have not forced pi to be a non-increasing function of b, because, as 
discussed above, the decrease of the upper conhdence limit when b increases is an undesirable 
feature from the physical point of view. Therefore, there is no reason to force pi to be a non¬ 
increasing function of b. The only ehect of this imposition is to introduce an overcoverage 
and then to increase the degree of conservativeness of the upper conhdence limits, which is 
an undesirable ehect. 

The decrease of the upper conhdence limit for a given n as the mean expected background 
b increases is an unavoidable feature of the unihed approach. However, I will show in the 
next Section that this decrease can be weakened with the choice of a new ordering principle 
for the construction of the conhdence intervals. 


II. A NEW ORDERING PRINCIPLE 


The ordering principle introduced by Feldman and Cousins [Q] is useful in order to 
guarantee an automatic transition from two-sided conhdence intervals to upper conhdence 
limits. However, it is not unique. Therefore, one can investigate the possibility to hnd a new 
ordering principle which minimizes the decrease of the upper conhdence limit for a given n 
as the mean expected background b increases. With this aim one can notice that the value 
( p..8D of Pbest(^^) decreases proportionally to n until it vanishes an then it is forced to be 
non-negative “by hand”. It is clear that it would be preferable to have a reference value for 
p which is automatically always positive. Such a positive reference value would imply that 
for small but not vanishing values of /i the values of n smaller than b have a high rank. As a 
consequence, the decrease of the upper conhdence limit for a given n as the mean expected 
background b increases is weakened with respect to the one obtained with the Feldman and 
Cousins ordering principle. 

The new ordering principle that I propose here is based on the choice for the reference 
value for /r of the bayesian expectation value: 


Pref(n) = / fiP{fi\n,b)dfi, 


( 2 . 1 ) 


where P(/r|n, b) is the bayesian probability distribution for p, calculated assuming^ a constant 


1 


This assumption is arbitrary, but it seems to be the most reasonable one if there is no prior 
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prior for /i > 0 (see, for example, 


-1 


F(/i|n, h) = {h + ^iYe ^ ^ 


k=0 


( 2 . 2 ) 


The result of the integral (|2.1|) calculated by parts is 


kb^ 


-1 


/iref(n) = ’^ + 1 - I ) [Y1 


vfc=0 


vfc=0 


k\ 


(2,3) 


The obvious inequality iniplies that Pref(^) > 1- Therefore, 

firef{n) represents a reference value for fi that not only is non-negative, as desired in order 
to have an automatic transition from two-sided intervals to upper limits, but is even bigger 
or equal than one. This is a desirable characteristics in order to obtain a weak decrease of 
the upper confidence limit for a given n when the expected background b increases. 

The implementation of the new ordering principle for the calculation of the confidence 
belt is identical to the implementation of the Feldman and Cousins ordering principle, except 
for the fact that in the calculation of the likelihood ratio R{n), which determines the rank 
of each value of n for a fixed value of /r, fihestin) is replaced by fj,rei{n): 


R{n) 


P{n\^i,b) 

P(n|jnref, h) ' 


(2.4) 


The resulting 90% CL confidence belts for 6 = 2, 3,4, 5 are shown in Fig. |^. 

Confronting Fig. ^with Fig. |l|one can see that the confidence belts are similar for n ^ b, 
but the new upper confidence limits for n < b are larger than the corresponding ones 
obtained with the Feldman and Cousins ordering principle. For example, four experiments 
expecting, respectively, 2, 3, 4 and 5 background events and observing nobs = 1 lead to 
the 90% CL upper confidence limits pi = 2.99,2.43,2.28,2.16, respectively, which can be 
confronted with the respective values pi = 2.53,1.88,1.34,1.20 obtained with the Feldman 
and Cousins ordering principle. It is clear that the decrease of pi as b is increased is milder 
with the new ordering principle. This is fully illustrated in Fig. ^ where I have plotted the 
upper end of the 90% CL confidence intervals obtained with the new ordering principle 
for n = 0,1, 2, 3,4, 5 as a function of the mean expected background b. One can see that for 
small values of b the decrease of the upper end fxi of the confidence intervals as a function of 
b is weaker than in Fig. and for large values of b the value of fii stabilizes around a value 
close to 1.7, which is higher than in Fig. |^. 

Since the new ordering principle gives upper bounds that are higher than those obtained 
with the Feldman and Cousins ordering principle, it must give also higher lower bounds. This 
is confirmed by a comparison of Figs. and ^ and is illustrated in detail in Fig. where the 
values of the lower confidence limit /i 2 obtained with the two methods for n = 3,..., 9 are 


information on the value of p, which is the parameter probed directly in the experiment. 
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plotted as functions of the mean expected backgronnd b. One can see that for each valne of 
n the maximnm valne of b for which the lower confidence limit is different from zero is the 
same in the two methods. This is a conseqnence of the fact (clearly seen with a comparison 
of Figs. |l| and |^) that the minimnm valne of n for which there is a lower bonnd different 
from zero is the same in the two methods (as must be, since in both approaches the lower 
limit of the acceptance interval for /r = 0 is at n = 0). Hence, the experimental possibility to 
claim a positive effect is the same in the two methods. Furthermore, Fig. ^ shows that the 
difference between the lower limits obtained with the two methods is moderate and tends 
to vanish for n S> & (indeed, for n S> & we have /iref — n = /Xbest and the two methods are 
practically eqnivalent). 


III. APPLICATION TO THE RESULT OF THE KARMEN EXPERIMENT 

In this Section I discuss the application of the new ordering principle proposed in Section 
|l|to the analysis of the recent null result of the KARMEN neutrino oscillation experiment [Q. 

The KARMEN experiment is searching for nentrino oscillations (see, for example, [Q) 
in the —>■ z/g channel with a sensitivity in the region of the nentrino mixing parameters 
sin^ 29 and Am^ which is allowed by the positive results of the LSND experiment . So far 
the KARMEN experiment measnred no events, with an expected backgronnd of 2.88 ± 0.13 
events [^. In this case the nnihed approach leads to an upper conhdence limit of 1.1 
events for the mean /i of nentrino oscillation events. On the other hand, the sensitivity 
of the experiment, dehned as “the average npper limit one conld get from an ensemble of 
experiments with the expected backgronnd and no true signal”, is of 4.4 events, fonr times 
bigger than the npper conhdence limit (notice that the average upper limit is calculated 
using the unihed approach). This is clearly a pathological case in which the observation of 
less backgronnd events than expected leads to a stringent npper conhdence limit /ii, even if 
the experiment is not sensitive to the corresponding small valnes of /i. 

The exclusion curves (the conhdence region lies on the left of the exclusion curves) in the 
sin^ 20-Am^ plane corresponding to the upper conhdence limit of 1.1 events obtained with 
the nnihed approach and to the sensitivity of the experiment (4.4 events) are reproduced in 
Fig. P (the solid curves passing through the hlled and empty circles, respectively). The solid 
curve passing throngh the hlled squares corresponds to the npper limit of 2.3 signal events 
obtained with the bayesian approach. The shadowed area in Fig. ^ is the region allowed 
at 90% CL by the resnlts of the LSND experiment 0 and the dashed, dash-dotted and 
dash-dot-dotted curves are the 90% CL exclusion curves of the Bugey 0, BNL E776 [§ and 
CCFR P experiments, respectively. 

The stringent limit for the nentrino mixing parameters represented by the nnihed ap¬ 
proach exclusion curve strongly depends on the fact that none of the expected background 
events has been observed. Indeed, the nnihed approach exclusion curve is based on the 90% 
CL exclusion of a mean signal of 1.1 events, bnt the signal that would have been exclnded if 
KARMEN had observed, for example, two (backgronnd) events wonld have been 3.1 events 
and the corresponding exclusion curve in the sin^ 26-^rn? plane wonld lie on the right of 
the bayesian exclusion curve. In other words, the fact that the nnihed approach exclusion 
curve is very stringent is not dne to the fact that no nentrino oscillations were observed, bnt 
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to the fact that no background event has been observed. This is clearly undesirable from 
the physical point of view. 

The new ordering principle proposed in the previous Section allows to obtain an exclusion 
curve with the correct coverage that minimizes the effect of the fact that no background 
event has been observed in the KARMEN experiment. The new ordering principle gives a 
90% CL upper conhdence limit of 1.9 events for the mean /r of neutrino oscillation events. 
This limit is close to the upper limit of 2.3 events obtained in the bayesian approach. As 
shown in Fig. the exclusion curve corresponding to the upper conhdence limit obtained 
with the new ordering principle (the solid curve passing through the hlled triangle) lies close 
to the bayesian exclusion curve. 

The sensitivity of the KARMEN experiment calculated with the new ordering principle 
is of 4.7 events, rather close to the sensitivity of 4.4 events calculated with the unihed 
approach. The corresponding curve is shown in Fig. ^ (the solid curve passing through the 
empty triangles). 

Looking at Fig. ^ one can see that the new ordering principle allows to obtain from the 
results of the KARMEN experiment an exclusion curve that is more reliable than the one 
obtained with the unihed approach, because the discrepancy between the exclusion curve 
and the corresponding sensitivity curve is smaller. Furthermore, since the new ordering 
principle implies an exclusion curve close to that obtained with the bayesian approach, we 
have a nice and desirable agreement of the results obtained with different statistical methods, 
one of which (the new ordering principle) guarantees a correct coverage. 

In conclusion of this Section, I would like to emphasize that, in spite of the improvement 
for the physical interpretation of the result of the KARMEN experiment obtained with the 
introduction of the new ordering principle, the fact that the KARMEN exclusion curves 
obtained with different statistical methods are signihcantly different shows that no hrm 
physical conclusion can be inferred from the null result of the KARMEN experiment. Since 
the KARMEN experiment is continuing to take data, some background events consistent 
with the expected rate should be observed in the near future and the different curves obtained 
with different statistical methods should converge. Only in that case it will be possible to 
conhrm or exclude the LSND indication in favor of neutrino oscillations. 


IV. CONCLUSIONS 

The new ordering principle for the classical statistical analysis of Poisson processes with 
background proposed here allows to improve the unihed approach recently proposed by 
Feldman and Cousins |^. 

The conhdence intervals obtained with the new ordering principle take advantage of the 
desirable features of the unihed approach and of the bayesian approach. The choice of the 
bayesian expectation value ( |2.1| ) as the reference value Hreiin) for the new ordering guarantees 
a reference value bigger or equal than one. The implementation of the new ordering principle 
in the same way as the Feldman and Cousins ordering principle is implemented in the unihed 
approach guarantees that the resulting conhdence intervals have the correct coverage and 
that there is an automatic transition from two-sided intervals to upper limits for n <h. The 
fact that the reference value /iref(^) is > 1 implies that the decrease of the upper conhdence 
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limit yUi for the mean signal n as the mean expected background b increases is weaker with 
the new ordering principle than with the Feldman and Cousins ordering principle. This is a 
desirable feature from a physical point of view, because it allows to minimize the influence 
that the observation of less background events than expected has on the upper conhdence 
limit for the mean true signal events. 

If one does not like the fact that in the new ordering principle the reference value 
has been obtained using the bayesian method, one can consider Eq. (|2.3|) as the formula that 
allows to dehne in order to obtain the desired result (as the conditions ( [I.5|) and (p..6|) 

allow to obtain central conhdence intervals and upper conhdence limits, respectively). After 
all, the construction of the conhdence belt, including the ordering principle, is not uniquely 
dehned from hrst principles and a scientist can choose the method which maximizes the 
scientihc meaning of the resulting conhdence intervals, as long as the correct coverage is 
guaranteed. 

In Section EJi have applied the new ordering principle to the analysis of the null result 
of the KARMEN neutrino oscillation experiment [Q. The new ordering principle gives an 
exclusion curve in the plane of the neutrino mixing parameters sin^ 29 and Am^ which lies 
close to that obtained with the bayesian approach and is signihcantly less stringent than 
the one obtained with the unihed approach. Since the stringency of the unihed approach 
exclusion curve is due to the fact that no background events have been observed (with 2.88 
mean expected background events) and not to the sensitivity of the experiment to a small 
true neutrino oscillation signal, I think that the exclusion curve obtained with the new 
ordering represents more appropriately the physical implications of the null result of the 
KARMEN experiment. 

Finally, I would like to notice that the new ordering principle has been formulated here for 
the case of a Poisson process with background, but can be generalized to other controversial 
cases, as that of gaussian errors with a bounded physical region 0. 
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FIGURES 


Fig. Confidence belts for 90% CL obtained with the nnihed approach for a Poisson 
process with backgronnd 6 = 2 (region between the two solid lines), 6 = 3 (region between 
the two dashed lines), 6 = 4 (region between the two dash-dotted lines), 6 = 5 (region 
between the two dash-dot-dotted lines). 


Fig. a Upper end of the 90% CL conhdence intervals obtained with the nnihed approach 
for n = 0,1, 2, 3,4, 5 as a function of the mean expected background 6. 


Fig. |. Conhdence belts for 90% CL obtained with the new ordering principle for a 
Poisson process with background 6 = 2 (region between the two solid lines), 6 = 3 (region 
between the two dashed lines), 6 = 4 (region between the two dash-dotted lines), 6 = 5 
(region between the two dash-dot-dotted lines). 


Fig. Upper end of the 90% CL conhdence intervals obtained with the new ordering 
principle for n = 0,1, 2, 3,4, 5 as a function of the mean expected background 6. 


Fig. ^ Lower end of the 90% CL conhdence intervals obtained with the nnihed approach 
and with the new ordering principle for n = 3,..., 9 (the solid line, ..., the dash-dot-dotted 
line) as a function of the mean expected background 6. 


Fig. §. 90% CL exclusion curves in the plane of the neutrino oscillation parameters 
sin^ 26'-Am^ corresponding to the null result of the KARMEN experiment 0. The solid 
curves passing through the hlled squares, circles and triangles are obtained with the bayesian 
approach, the nnihed approach and the new ordering principle, respectively. The solid curves 
passing through the empty circles and triangles are the sensitivity curves obtained with the 
nnihed approach and the new ordering principle, respectively. The shadowed area is the 
region allowed at 90% CL by the results of the LSND experiment P] and the dashed, dash- 
dotted and dash-dot-dotted curves are the 90% CL exclusion curves of the Bugey [^, BNL 
E776 P and CCFR 0 experiments, respectively. 
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